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ABSTRACT 

This research follows on previous studies that investigated how digitally augmented devices and knowledge scaffolds 
enhance learning in a science museum. We investigated what combination of scaffolds could be used in conjunction with 
the unique characteristics of informal participation to increase conceptual and cognitive outcomes. 307 students from nine 
middle schools participated in the study. Six scaffolds were used in varying combinations. The first was the digital 
augmentation. The next five were adaptations of knowledge-building scaffolds. Results demonstrated greater cognitive 
abilities in terms of theorizing can be obtained with particular combinations of scaffolds, however, there appears to be a 
trade-off in what students can learn conceptually in informal environments in which learning takes place in short 
episodes. 
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1. INTRODUCTION 

This research seeks to understand how aspects of knowledge building pedagogy (Scardamalia, 2002; 
Scardamalia & Bereiter, 2006), known to be successful in formal classrooms, might be adapted to reach 
broader informal audiences to improve scientific literacy in museums. The recent National Research Council 
(NRC) report on learning science in informal environments (NRC, 2009) highlights the potential of non- 
school settings, such as museums, for engaging large populations in scientific investigation. Despite this 
potential, the NRC report outlines the need for essential research in three key areas. First, while there is 
ample evidence that suggests informal environments increase engagement and interest, fewer studies have 
focused on how those experiences result in conceptual gains of science content. Second, in terms of scientific 
skills, designed interactives have been shown to increase lower level skills such as manipulating and 
observing, however more challenging skills such as critical thinking and theorizing are less frequently 
demonstrated. Finally, as digital platforms are increasingly incorporated in informal settings, more research is 
needed to determine how they enhance the learning experience. 

The research reported here follows on a series of studies that investigate how digitally augmented devices 
and knowledge-building activities can enhance learning in a science museum (Yoon et ak, in press-a; in 
press-b). In the series we use a quasi-experimental design in which students in multiple conditions interact 
with a museum device using digitally augmented information and varying arrangements of knowledge 
scaffolds. Results thus far have suggested that digital augmentations can help in conceptual development and 
that students’ abilities to interpret information about a phenomenon using knowledge-building scaffolds can 
improve cognitive understanding in some conditions. However, one major concern that has emerged in our 
observations of student interaction in the more highly scaffolded condition is the over-formalization of the 
learning experience that appeared to be needed to achieve those results. Specifically, when students were 
asked to use learning scaffolds that were delivered in, for example, worksheets with knowledge prompts, 
students were less explorative and referred to the scaffolds to dictate their next steps. Students in less 
formally structured conditions were generally more playful and experimented on their own — much like 
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students would normally behave in museum exhibits during a school field trip. Informal learning 
environments can enhance interest and engagement because the experience does not look like school, i.e., it 
is fluid, sporadic, and participant-driven (Falk & Dierking, 1992, 2000). The research in this study aims to 
investigate the issue of over-formalization. Specifically, the research question we seek to answer is: Which 
combination of learning scaffolds can optimally be used in a science museum taking into consideration the 
unique characteristics of informal participation to increase learning outcomes? 


2. BODY OF PAPER 

2.1 Theoretical Considerations 

2.1.1 Augmented Reality as a Scaffold for Learning 

In the most recent Horizon Report, the New Media Consortium (2012) discusses the enormous potential 
augmented reality (AR) capabilities have on learning and assessment in enabling people to construct new 
understanding. AR experiences layer digital displays over 3D real world environments (New Media 
Consortium, 2012) providing access to normally hidden data that users can use to develop deeper knowledge 
about a phenomenon or take a different immersive perspective to broaden their understanding. In the past 
decade, practical uses of AR have emerged in fields such as games, marketing and advertising, films, 
navigation, and for medical and military applications (El Sayad et ak, 2011). Although newer in education, 
over the last few years, there have been studies that illustrate AR’s potential for learning particularly in the 
field of science education. For example, Dunleavy and colleagues (2009) document high student engagement 
influenced by the ability to collaboratively problem solve and collect data in the real world in their Alien 
Contact! handheld AR environment. Squire and Klopfer (2007) detail the impact of their AR game 
Environmental Detectives on accessing student prior knowledge by connecting academic content to physical 
spaces that students are familiar with. In Outbreak @ The Institute, Rosenbaum and colleagues (2007) 
document the affordance of their augmented reality game play to include authentic scientific inquiry and 
understanding the dynamic nature of system interactions. In these studies, the indirect correlates of student 
learning, i.e., engagement, prior knowledge, and processes in scientific practice, are important outcomes of 
the research and provide valuable impetus for pursuing further studies on what and how students learn in 
terms of scientific knowledge. 

AR technologies have also been incorporated in museums to enhance visitors’ experience by improving 
their interest, engagement, and access to information (Baber et ak, 2001; Damala et ak, 2008; Hall & 
Bannon, 2006). For example, Szymanksi and colleagues (2008) examined how two different electronic 
guidebook prototypes affected visitors’ social interaction. As visitors toured a historic house, the guidebooks 
provided information about the historic artifacts that the visitors encountered. Results indicated that these 
guidebooks led visitors to engage in more content-rich discussions with each other. Furthermore, their 
exploration of the room and its objects were enhanced (Szymanski et ak, 2008). Similarly, Baber et ak 
(2008), in testing three different AR platforms in an art museum, discovered that different devices elicited 
varying lengths of visitor viewing times and visitor preferences. Damala et ak (2008) also tested an AR- 
enabled mobile multimedia museum guide in a fine arts museum and found that visitors enjoyed the playful 
content presentation that the museum guide enabled. Finally, Hall and Bannon (2006) investigated the 
effects of a digitally augmented exhibit on children. When children were given RFID sensors that could 
detect exhibit locations and unlock virtual information, their interest and engagement increased. 
Collectively, these studies demonstrate the impact that AR can have on visitors in a museum setting in terms 
of increasing engagement, interest, and access to information through scaffolded experiences. 

Encouraged by this previous research and in envisioning potential extended uses of AR in museums as 
scaffolds for knowledge improvement, the present study addresses the real need for learning research (NRC, 
2009) in informal environments through connecting with the notion of scaffolds. In the next section, we 
describe our rationale underpinning the use of scaffolds and in particular the use of knowledge-building 
scaffolds to improve learning. 
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2.1.2 Learning through Knowledge-Building Scaffolds 

The use of scaffolds in educational technology applications has been researched fairly extensively to support 
scientific inquiry and cognitive tasks (e.g., Quintana, 2004). In particular, a long-standing program of 
research in the learning sciences that is premised on designing learning environments through the intentional 
application of technological and pedagogical scaffolds is knowledge building (Yoon, 2008; Bereiter, 2002; 
Scardamalia, 2002; Scardamalia & Bereiter, 2006). This approach is centrally focused on the goal of 
improving ideas in the same way that knowledge work is done by experts in real world contexts (Scardamalia 
& Bereiter, 2006). Primarily applied in school classrooms, knowledge -building studies have been shown to 
increase student scientific abilities in explanation, interpreting and evaluating information, and knowledge 
advancement (van Aalst, 2009). Students also acquire deep theoretical understanding of scientific 
phenomenon through collective sustained inquiry and research on problems that can range from what causes 
leaves to change color in the fall in a grade 1 classroom (Scardamalia, 2002) to the complex influences on 
genetic engineering research with middle and high school students (Yoon, 2008; 201 1). 

The technological application. Knowledge Forum and associated pedagogy use educational scaffolds to 
enable public, collective contributions that shape the knowledge constructed in the learning community. Such 
scaffolds include prompts for consensus building, generalizations, differentiation between evidence and 
theories, and peer evaluation. For example, a prompt such as “My theory is..." encourages students to use 
evidence to construct a more general understanding of a class of scientific phenomena. Similarly, students 
can create a “rise above” note, enabled by the archived database of peer exchanges, which is a distillation of 
an idea or theory from a collection of previous peer exchanges that provide students with opportunities to 
think across diverse perspectives and to arrive at conclusions about how the collective learning community 
views a scientific issue (Yoon, 2008). 

Collaboration also factors prominently into the knowledge-building approach. By working with others 
discursively to problem solve, evaluate evidence, and identify important shared understanding , students are 
able to more deeply reflect on what they know rather than learning independently, or learning through textual 
modes. This decentralized, public, and distributed participation promotes what Scardamalia (2002) calls 
collective cognitive responsibility where the impetus for learning is generated by consensus within the 
community rather than by the teacher. From this set of theoretical and pedagogical descriptions, our series of 
studies uses varying degrees of what we collectively refer to as knowledge-building scaffolds which include: 
knowledge prompts, a bank of peer ideas, working in collaborative groups, instructions for generating 
consensus, and worksheets for recording shared understanding. However, because knowledge building 
requires the development of a community with shared understanding, language, and goals, learning events 
evolve over longer periods of time than informal environments may afford. Van Aalst (2009) characterizes 
learning experiences that are less focused on the community as knowledge construction in which students 
may collaborate in small groups on tasks that require less synthesis and reflection on the knowledge 
advancement process. We have understood the limitation of our informal setting and population in terms of 
achieving a true knowledge building community in previous studies (e.g.. Author, in press-b) but have 
nevertheless attempted to investigate how aspects of knowledge building pedagogy can be applied in 
informal environments given its success in formal classrooms. The issue of over-formalizing the experience 
was a phenomenon that emerged as an important context-dependent factor in considering how informal 
learning takes place (Author, in press-b), which is the subject of this study. In the next section, we describe 
the features of informal environments that make learning unique. 

2.1.3 Features of Informal Learning Environments 

Learning in informal spaces is fluid, sporadic, social, and participant driven — characteristics that contrast the 
highly structured formal classroom experience (Honey & Hilton, 2011, NRC, 2009, Squire & Patterson, 
2009). Activities are often experienced in single -visit episodes (Falk et ak, 2007) where visitors learn on their 
own with little structured follow-up or reflection. McManus (1994) has characterized typical visitors as 
demonstrating scouting behaviors within museum exhibits, where they roam around, encounter devices, and 
act quickly to discover the intended information. Thus, more systematic learning studies are difficult to 
design. However, science museum exhibit developers do intentionally design learning spaces that mix a 
variety of supports for learning. For example, exhibit devices (individual interactive kiosks within the larger 
exhibit) are deliberately grouped and arranged to encourage progressive engagement with topics (Grinell, 
2006) in support of the exhibit’s overall learning objective. Many devices are accompanied by posted graphic 
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panels (Serrell, 1998, 2006) that provide printed content in order to support the interpretation of scientific 
phenomena. Finally, visitors often encounter exhibit educators (Serrell, 2006) who are skilled at facilitating 
learning in social groups around exhibit devices. They spark conversation, notice ideas, and encourage 
theorizing. We see parallels between these aspects of the informal learning environment and the structure of 
effective knowledge building in classroom settings. 

For this study, we deliberately attempted to construct conditions that probe these parallels. The 
application of a digital augmentation to an exhibit device provides a first layer of interpretive support and 
acts as a primary scaffold. We then added posters and worksheets with progressively more rigid layers of 
structure around the experience to advance the scaffolding for analysis. In some conditions, we prompted or 
required group work as a strategy for measuring the benefit of collaboration. We attempted to detect the 
frequency of theory building and, in some conditions, prompted it explicitly. As our methods and results will 
show, these scaffolds had varying levels of success. 

2.2 Methods 

2.2.1 Participants and Context 

For our study population, we recruited teachers to bring their students who had previously participated in 
workshops and other teacher events at the museum or were referred to us to participate in the study by those 
teachers. In total, 307 students (52% female, 48% male) on a school fieldtrip from nine local middle schools 
(grades 6-8) participated in the study. We chose the middle grades for this study to have confidence that the 
students were developmentally able to theorize. Also, the study engaged students with the topic of electrical 
conductivity. By grade six, all students would have encountered the topic in their classrooms, as dictated by 
the local standardized curriculum. This provided a common ground for the participants, which increased the 
chances that they would have similar prior knowledge. 

An exhibit device called “Be the Path” provided the context for the study (see Figure 1). The device 
consisted of two metal spheres on a table, approximately one foot apart, with one connected by a wire to a 
battery and the other connected to a light bulb. The instructions on the device provided little direction, simply 
suggesting “try to complete the circuit.” The students attempted different configurations to complete the 
circuit and light the bulb. Once the circuit was completed, a projected visualization of electrons flowing 
around the complete loop appeared. A more complete description of the technological platform and how we 
developed it can be found in (Author, in press-b). 



Figure 1. “Be the Path” device with digital augmentation 


Students were randomly assigned to conditions, constructed to represent increasing use of scaffolds. Six 
scaffolds were used in varying combinations. The first was the digital augmentation. The next five were 
adaptations of knowledge-building scaffolds which included: 1) group work; 2) worksheet questions 
directing student attention to relevant information; 3) directions about how to work together; 4) embedded 
worksheet knowledge prompts; and 5) bank of previous worksheet responses. Condition 1 (Cl) served as the 
control group with no scaffolds. Condition 2 (C2) represented the digitally-augmented device with no 
additional scaffolds. Condition 3 (C3) was the same as C2 but with posted worksheet questions such as What 
happened when you touched both metal spheres?'. What happened to make the bulb light up? Students in Cl- 
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C3 used the device individually and completed individual worksheets afterwards. Condition 4 (C4) students 
worked in groups, with posted worksheet questions, completing the worksheet individually after the 
experience. C4 was meant to represent the typical informal learning state in which groups of students 
encounter posted labels with questions about devices. Condition 5 (C5) students had the same treatment as 
C4 students with additional knowledge prompts. Knowledge prompts included. Our hypothesis is'. Our 
Theory is', and Others have said. A bank of other student worksheet responses was also posted for students to 
refer to as they answered the worksheet questions. They completed the worksheet collaboratively after the 
experience with one worksheet per group. Condition 6 (C6) students had the same treatment as C5 except the 
worksheet was completed collectively during the experience. Table 1 shows the number of students who 
participated in each condition. 

Table 1. Number of students who participated in each condition 


Condition 

Number of Students 

1 

30 

2 

31 

3 

31 

4 

24 groups of 3 (72) 

5 

23 groups of 3, 1 group of 2 (71) 

6 

24 groups of 3 (72) 


2.2.2 Data Sources and Analysis 

Three data sources were collected and analyzed through a quasi-experimental mixed-methods approach: 
surveys, worksheets, and interviews. Each data source is detailed below. Surveys : A conceptual knowledge 
survey was administered to students in each group before and after the intervention. The survey posed five 
general multiple-choice content questions each valued at one point, related to the scientific topic of electrical 
conductivity and circuits. An additional open-ended question on the survey also solicited responses that 
demonstrated knowledge directly related to the device experience, i.e., “Think about an electric circuit that 
supplies electricity to a light bulb. What parts make it work so that the bulb lights up?” Responses to this 
question were coded on a five-point Likert-scale from no understanding (0) to complete understanding (4). 
For the coding scheme, the dataset was reviewed qualitatively by members of the research team in a series of 
data analysis sessions during which responses were compared to answers in a teacher’s manual. We also 
consulted a physicist on staff at the museum to help the team design the scoring rubric. Once codes were 
established, a categorization manual was constructed. Two graduate students, who were not otherwise 
familiar with the research project, were trained on the coding scheme, and inter-rater reliability was obtained 
on 20% of the data with greater than 90 percent agreement. Collectively, the maximum possible score on the 
conceptual knowledge survey was nine points. A paired-samples t-test was conducted to determine statistical 
significance in conceptual knowledge gain within each condition after treatment. A between-subjects 
ANCOVA was conducted to determine whether the main condition effect was significant. For this survey, we 
weren’t able to conduct a multilevel analysis due to the small sample size but we are aware of the non- 
independence issues that can potentially impact results. However, as Cress (2008) notes, computer supported 
collaborative learning environments are designed to be influenced by group interactions and we intentionally, 
designed the conditions to understand the impact of learning scaffolds of which group interaction is one. 
Worksheets : A worksheet was used to gather data from all participating students, regardless of condition. 
Students in Conditions 1, 2, and 3 completed the worksheet individually immediately after they finished 
interacting with the device. They moved away from the device to be seated at tables for the task. Students in 
Condition 6 actually completed the worksheet during the group experience with a single worksheet on a 
clipboard being used for their collective response. Each group of three was allowed to negotiate the work 
process for themselves. For example, some groups elected a single scribe while others passed the sheet 
around to share the scribe duties. The worksheet required them to respond to this open-ended question: 
“What are you supposed to learn by using this device?” This question was intended to elicit responses that 
demonstrated ability to theorize from the interaction with the device, i.e., understanding of electrical circuits 
and how the human body functioned as a conductor. Responses were coded on four levels from no 
understanding (0) to complete understanding (3). Categorization manual construction, coding and reliability 
activities similar to those performed for the open-ended conceptual knowledge question were conducted. An 
ANOVA was conducted on the worksheet data set to determine whether there was a statistically significant 
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difference in responses between the conditions and a post-hoc Tukey HSD comparison was conducted to 
determine the source of the difference. Interviews : In order to investigate how knowledge-building scaffolds 
impacted the nature of the intervention, 66 students (-20% of the population) from C5 and C6 were randomly 
selected for short interviews to determine the utility of the scaffolds following their interaction with the 
device. We asked the students to evaluate each of the scaffolds they encountered in order to understand 
which were more or less impactful on their experience. 


2.3 Results 

2.3.1 Conceptual Knowledge 

Across the conditions, pre- and post-survey means ranged between 2.903 (pre-survey, C3) to 4.014 (post- 
survey, C4) out of a possible score of 9. Table 2 shows the results of a paired-samples t-test conducted within 
each condition. The table shows that gains in all conditions except Cl were statistically significant. Students 
in C4 demonstrated the greatest gains, with students in C3 a close second. However a between-subjects 
ANCOVA using the pre-test scores as covariate showed that there was no significant difference in the 
outcomes between six groups, F(5, 300) = .546, p=.741. 

Table 2. Results of Paired-Samples T-Test Comparing Means within Conditions 


Condition 

Mean Difference 

SD 

t 

df 

Sig. 

(2-tailed) 

1 

.337 

1.691 

1.187 

29 

.245 

2 

.484 

1.313 

2.051 

30 

.049* 

3 

.677 

1.833 

2.058 

30 

.048* 

4 

.681 

1.362 

4.241 

71 

.001* 

5 

.366 

1.355 

2.278 

70 

.026* 

6 

.472 

1.538 

2.605 

71 

.011* 


* p<0.05 


2.3.2 Cognitive Understanding 

Students were asked What are you supposed to learn by using this device? Response scores ranged from 
2.233 (low emergent understanding) to 2.800 (nearly partial understanding). Figure 2 shows an increasing 
trend from Cl to C6 in the means of theorizing abilities. 


Worksheet Responses: Ability to 



Cl C2 C3 C4 C5 C6 

Condition 


Figure 2. Mean scores for ability to theorize for conditions 1-6 

An ANOVA comparing mean scores showed a significant difference in results, F(5)=2.234, p=.Q52. A 
post-hoc Tukey analysis attributed the difference to the higher mean of C6 which was marginally 
significantly higher than Cl (p=.056). Differences between all other conditions were statistically not 
significant. 
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2.3.3 Impact of Knowledge-Building Scaffolds 

C5 and C6 students were asked which knowledge-building scaffolds were helpful. Figure 3 graphs their 
responses. 


Frequency of student Responses 
to Scaffold Helpfulness (n=66) 


65 



Figure 3. Frequency of student responses indicating helpfulness of scaffolds 

Many students found both group work and the worksheet helpful while fewer than 1/2 found the 
directions on how to work together helpful. Greater than 2/3 indicated that the knowledge-building prompts 
were helpful and greater than 1/2 indicated that the bank of previous answers was helpful. 


3. CONCLUSION 

Based on the conceptual gains shown in Table 2, this study supports previous findings (Yoon et ah, in press- 
a; in press-b) that digital augmentations can serve as valuable scaffolds for conceptual learning. Furthermore, 
where C3 and C4 students had the highest mean difference, we found that worksheet questions, which we 
consider to be a form of graphic panel (Serrel, 2006) when posted, can also support conceptual gains. We 
also found this to be true in our previous studies. When interviewed, C5 and C6 students said that working in 
groups was the most helpful scaffold besides worksheets, which suggests that the combination of group work 
and posted worksheets can assist in learning science concepts. 

Although the knowledge prompts and the bank of previous answers were selected fewer times as helpful 
scaffolds than working in groups and worksheets, the fact that a good portion of the population endorsed 
these more formal scaffolds is encouraging. Also, the result that C6 students had the highest score in higher- 
order reasoning somewhat supports an argument for their continued use. However, the main reason for 
undertaking this research was to understand the extent to which knowledge-building scaffolds enhance the 
museum learning experience and which are most impactful. We hypothesized that “over-formalization” in an 
informal environment may negatively impact learning. This would include the use of scaffolds other than the 
kinds of graphic panels that are normally encountered. Our results indicate that greater conceptual gains 
occurred when scaffolds were used in less formal ways, i.e., when the worksheet questions were posted (C3 
and C4) and when they looked more like graphic panels typically found in museum settings, which in part 
satisfies our hypothesis. The fact that C6 students demonstrated greater theorizing ability suggests, however, 
that there may be a trade-off in what students can learn in informal environments in which learning takes 
place in short episodes. That is, we may be able to improve student’s conceptual understanding somewhat at 
the expense of improved cognitive gains if we consider the unique characteristics of informal settings. 

Repeat visitation and use of an exhibit space could counteract this pattern as conceptual understanding 
can grow over time and later become more formalized for the sake of cognitive gain. Through repeated use of 
an exhibit device, especially with a variety of knowledge -building scaffolds available, students find multiple 
learning pathways. The habit of repeated use of an exhibit remains a challenge given the episodic nature of 
visitation. On possible way to address this issue is a distributed virtual social learning space that can extend 
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engagement with topics from multiple exhibit devices outside of the museum’s physical space. Such a 
collaborative space is currently in development for testing in future phases of our research. 
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